User-centred adaptive spoken dialogue modelling

This dissertation investigates novel concepts and methods for the automatic recognition of dynamic user properties using statistical supervised learning and for the integration of these properties into the dialogue management process. The aim is to model the course of the dialogue adaptive to the user. Current commercial spoken dialogue systems usually do not account for dynamic user properties like the user’s satisfaction level. Even in state-of-the-art research systems, this type of adaptation is usually missing. However, if the system was aware of these properties, it would be able to have a better understanding of the current situation and thus to react more appropriately. Therefore, the goal of this work is to introduce this type of user-centred adaptation by separating the problem into two sub-problems: recognising the user state, i.e., the dynamic user properties, and integrating its estimation of the user state into the dialogue management process. Before these individual sub-problems are approached, first, the necessary background, which is important for understanding the content of this thesis, is described containing relevant information about spoken dialogue systems and supervised machine learning. Following this description of the background, research of others which aims at solving similar research problems is described including a clear distinction of their work to ours. For the first sub-problem of deriving the user state, we consider four different user states: the user satisfaction (US), the perceived coherence of the system reaction, the emotional state, and the intoxication level. Automatic recognition of these user states is based on supervised statistical learning. As US is a universal property, an emphasis is put on its automatic recognition with a focus on how temporal information may be used for this recognition process. Here, we propose three novel approaches on how to improve the US recognition performance: introducing an error correction module into the classification process, exploiting temporal learning algorithms by using a modified Markovian model, and optimising the feature set used as input to statistical classification models. While all of our proposed approaches result in a significant performance improvement, the best performance boost is achieved with an optimised feature set. With this feature set, we are able to improve the performance achieving a high correlation. Research on the recognition of the remaining three user states have also resulted in significant contributions to the state-of-the-art. For the automatic recognition of the perceived coherence of the system reaction, we are to our knowledge the first to connect aspects of the interaction with coherence. Exploiting this relationship, the problem is modelled as a statistical classification task. In order to improve the performance of speech-based emotion recognition, we propose two